Systems and methods for data transmission and rendering of virtual objects for display

ABSTRACT

System and methods are disclosed for displaying real-time, geo-registered data in an augmented reality or other virtual environment. The system and methods may be configured to supply missing information from the real-time, geo-registered data set in order to facilitate the display in three dimensions.

PRIORITY

This application claims priority to U.S. Provisional Application62/734,907, filed Sep. 21, 2018, and incorporated by reference herein inits entirety.

BACKGROUND

Although Microsoft's HoloLens is today's most capable commercialaugmented reality (AR) headset device, the current generation andsimilar high-end AR devices lack global navigation satellite systems(GNSS) as well as any understanding outside its local X, Y, Z referenceframe. This limits its functionality for outdoor use asLatitude/Longitude/Altitude information is unavailable.

Even if an AR device were to include GNSS, it would only provide theuser a position of the device, and not further information to relate thevirtual environment and the physical environment, such as direction,scale, landscape, elevation, etc.

The world is not a flat environment. It is not even a round environment.Instead, it is an oblique spheroid that includes modulations inelevation such as in valleys, mountains, hills, crevices, etc.Therefore, rendering virtual objects over a physical environment can bechallenging.

Conventionally, applications for augmented reality systems simplyoverlay static icons at static locations away from the user. The overlaydoes not permit virtual objects rendered in the augmented realityenvironment to correlated or positioned relative to the real world,and/or physical locations. Conventional systems therefore do not permitexternal information or data to be rendered in a virtual overlay createdwithin the physical environment in which the augmented environment ispositioned. At best, some systems may use markers, such as a QR code),in which to determine a location for a rendered virtual object. In thiscase, the virtual object is rendered relative to the marker, such as theQR code. However, such systems still do not account for the variation ofthe physical environment in which the marker and/or virtual image is tobe used. As in any conventional system, the rendering of the virtualobject is in a static/predetermined location relative to a systemparameter, whether it is to a detected object, position on the display,etc. These systems also cannot provide missing information or handlelarge data transfers for rendering objects in real time.

Rendering a data set within an augmented environment presents technicalchallenges and limitations. For example, high-resolution detailedelevation data is generally not available on the wearable device. Ifthis data is not present on the device, 2-D data sets cannot be properlyreferenced into the 3-D display frame with only latitude and longitudepoints. To illustrate, take a conventional dataset for positioningobjects within an area providing latitude and longitude for the object.If the location and direction of the headset were known, the objectsrelating to the dataset can be rendered in relation to the headset.However, their position in a real world overlay would be difficult. Ifthe data set is limited to latitude and longitude, then the elevationposition of the rendered object may be unknown. If all virtual objectsare rendered at the same elevation, while the physical terrain is atdifferent elevations, the experience may be illusory, the perception ofthe virtual objects may be distorted, and the augmented experienceincomplete.

Even the most modern mobile devices have limited power and onboardprocessing capabilities. Depending on the resolution, elevation datasets can be very large, creating data download, local storage andprocessor loading challenges.

Large elevation data sets can require extensive processing power torender and may present a problem for certain wearable devices. If thedisplay program has an option for different world scaling, each scaledversion will require additional processing of the elevation data set.These activities can quickly consume the wearable device's resources andbattery power reserves.

When accessing different data sources, the realities of differentcoordinate datums arise. Not all data sets utilize the same datum (i.e.WGS84, etc.) and to enable proper data exchange, it may be necessary forthe mobile device to re-project one (or more) data sets into a commonframe of reference. Failure to do so could result in incorrectassignment of target locations due to differences in the native dataset's frame of reference.

SUMMARY

Exemplary embodiments described herein include software solutions foruse with computer displays and/or augmented reality and/or virtualreality systems to generate a “geo-capable” device that functions in aworld-wide reference frame and understands where the user is looking,thereby allowing the real-time display of georegistered information.

Exemplary embodiments may include a commercial AR software solution thatdelivers real-time, georegistered information. This novel capabilityallows the user to view digital holographic information, displayed inthree dimensions (3D), projected on the physical environment incoordinates referenced by Latitude, Longitude and Altitude. Exemplaryembodiments may be hardware agnostic and deployable to suitably equippedaugmented reality (AR) and/or virtual reality (VR) systems. Displayingreal-time, georegistered data in AR/VR reveals detailed informationabout the local environment and may enhance the user's overallsituational understanding.

Exemplary embodiments may include an intelligent approach to deviceresource and data management. Exemplary embodiments may thereforetransform and simplify data sets to maintain appropriate fidelity, whileminimizing the required data. Exemplary embodiments may thereforeinclude rendering data sets at the minimum allowable resolution to helpmitigate issues in processing power, processing time, power consumption,and combinations thereof. The data set resolution may, or may not,correspond to the maximum possible resolution based on the source dataavailable.

Exemplary embodiments may be used to determine a position of anelectronic display device in order to relate virtual objects to physicallocations in order to properly overlay, relate, and correlate virtualobjects with real world physical locations. Exemplary embodiments may beused to determine a position of an electronic device for creatingvirtual objects. Exemplary embodiments may be used to allow theestimation of location or direction of the electronic device, such asdetermine the position and direction in which a user is looking.Exemplary embodiments may be used to create the digital earth map onwhich the user is working. Exemplary embodiments may be used withelectronic devices with or without their own or in communication withtheir own positioning systems, such as a global navigation satellitesystem (GNSS).

FIGURES

FIGS. 1A-1C illustrate an exemplary comparison of a physical view of anarea and a view of the area augmented by virtual representations andinformation corresponding thereto correlated to the physicalenvironment.

FIG. 2 illustrates another exemplary virtually augmented realityenvironment in which representations of aircraft landing in the physicalenvironment of the augmented reality system are illustrated and overlayin relation to the physical environment.

FIG. 3 illustrates an exemplary flow diagram for methods and/or systemalgorithms according to embodiments described herein.

FIGS. 4A-4D illustrate a mesh overlay in which a data set of elevationis rendered on a physical environment.

FIG. 5 illustrates an exemplary data processing representation for theback end and real time processing according to embodiments describedherein.

FIG. 6 illustrates an exemplary system configuration according toembodiments described herein.

FIGS. 7A-7C illustrate an exemplary embodiment of a user display of anediting suite that enables integration of precision geo-registered 3Dinformation into video for a Utility Inspection application.

FIG. 8 illustrates exemplary embodiments used to show future buildinglocations and project phases.

FIG. 9 illustrates an exemplary overview display configuration accordingto embodiments described herein.

DETAILED DESCRIPTION

The following detailed description illustrates by way of example, not byway of limitation, the principles of the invention. This descriptionwill clearly enable one skilled in the art to make and use theinvention, and describes several embodiments, adaptations, variations,alternatives and uses of the invention, including what is presentlybelieved to be the best mode of carrying out the invention. It should beunderstood that the drawings are diagrammatic and schematicrepresentations of exemplary embodiments of the invention, and are notlimiting of the present invention nor are they necessarily drawn toscale.

Exemplary embodiments may present real-time, real-world data sets tousers in a hands-free, eyes-out interface. GIS data, video sources,LiDAR files and similar 3D data sets can be accurately rendered in thereal-world location via exemplary embodiments of the geopositioningtechnology, systems, and methods described herein.

Exemplary embodiments may include a system to integrate 3-D LiDAR dataand existing 2-D map tiles to create virtual object rendering in a threedimensional augmented reality environment. This capability may provideAR/VR-headset users easy visualization of combined 2-D/3-D/imagerydatasets. Exemplary embodiments may be configured to handle disparitybetween data set datums, limits of headset processing power, acceptabledata set reduction for large point-cloud clusters, video displaylimitations and managing battery life, and combinations thereof.

FIGS. 1A-1C illustrate an exemplary comparison of a physical view of anarea and a view of the area augmented by virtual representations andinformation corresponding thereto correlated to the physicalenvironment. FIG. 1A illustrates an exemplary physical environment. FIG.1B illustrates a virtual rendering of a data set overlayed on thephysical view looking in a first direction, and FIG. 1C illustrates avirtual rendering of the data set overlayed on the physical view lookingin a second direction.

As illustrated in FIGS. 1A-1C, the physical environment 100 includes aground layer 102 that defines different elevations. For example, a firstelevation 104 may be lower than a second elevation 106, such as createdby a hill, hole, or other topography. The physical environment 100 mayalso include physical objects 108.

As shown in FIGS. 1B and 1C, the system is configured to generatevirtual objects for rendering in an augmented or virtual environment andposition the virtual objects in physical relation to the physicalenvironment in which the system is augmenting. For example, as seen inFIGS. 1B and 1C, virtual objects 110 in the form of cubes are renderedin the location of physical buildings that existed before a fire.

FIG. 2 illustrates another exemplary virtually augmented realityenvironment in which representations of aircraft landing in the physicalenvironment of the augmented reality system are illustrated and overlayin relation to the physical environment. As shown, additionalinformation from or correlated to a data set may also be illustrated orvirtually rendered in the environment. The correlated information maycome from the data set itself or may be looked up through one or moreother available datasets, linked by information provided or related tothe data set.

FIG. 3 illustrates an exemplary flow diagram for methods and/or systemalgorithms according to embodiments described herein. As shown, thesystem may receive one or more source data sets. For example, the sourcedata sets may be map locations including latitude, longitude, andelevation. The data sets may correspond to any attribute of a physicalenvironment in which to create a reference frame.

Exemplary embodiments include systems and methods permitting thearbitrary conversion of a local augmented reality environment referenceframe to a world based reference frame in a unidirectional (in eitherdirection) and/or bidirectional conversion processes. For example, datasets represented in a physical environment latitude and longitude can berendered in relation to the x, y, z coordinate system of a headset orAR/VR environment and/or vice versa.

Exemplary embodiments described herein provide systems and methods foron the fly data processing and conversion. Exemplary embodiments mayprovide methods for optimizing the speed and processing of the methodsdescribed herein to reduce processing power, reduce power consumption,minimize time elapse for processing, and combinations thereof.

As seen in FIG. 3 , the system algorithms and methods may include one ormore source data corresponding to the physical environment, which may becombined and processed according to embodiments described herein.Exemplary embodiments result in a mesh representation of the physicalenvironment. The system algorithms and methods may then use the mesh,along with other source data for generating and rendering the virtualobject over the physical environment.

At steps 302A, 302B, one or more source data sets are received by thesystem corresponding to one or more attributes of the physicalenvironment. Two data sources are illustrated, but any number ofdatasets 302 n may be used. The source datasets may be in differentforms. For example, the data sets may correspond to different referenceframes, different scales, different units, different resolutions, andany combination thereof. Any number of datasets may also be provided. Inan exemplary embodiment, source data may be received from both publicinternet servers as well as private local storage, and locally scanneddata provided by the user for their specific application may also beused as exemplary data sources according to embodiments describedherein. Exemplary embodiments may also include internet of things orother nodes, sensors, or other sources as data sources according toembodiments described herein. For example, real time sensors, such aslocal temperature or pressures sensors, may provide real time conditiondata to the system as a data source. Source data may be in the form ofraster elevation data, LiDAR data, vector data, image data, text data,video data, and combinations thereof. The data may be provided asone-dimensional, two-dimensional, three, four, or five dimensionalformats. For example, a one dimensional data set may simply be readingsfrom a sensor without regard to a location or time information. Twodimensional data may include the data information with a position and/ortime element.

At step 304, the system analyzes, transforms, and/or combines, thereceived data from steps 302A-302 n. Exemplary embodiments may includesteps for converting data sets for uniformity and processing, combiningdifferent data sets, preparing the data sets for rendering in anaugmented or virtual reality environment, or any combination thereof.

In an exemplary embodiment, the one or more data sources may includedifferent information. For example, the information may be on differentscales, units, reference or coordinate systems, etc. The system mayrecognize the data configuration and/or receive information about thedata as an input and/or already have the data configuration programmedrelating to a source of the data. The system may take the raw data setand the data configuration to convert the data into a uniform structure.

In an exemplary embodiment, the datasets may be augmented to includemissing information. For example, if a dataset is lacking specificinformation, the system may generate the missing information byapproximating known information. The system may also, or alternativelyretrieve another data set that includes the missing information orinformation related to the missing information to generate the missinginformation. The system may therefore recognize a dataset is missinginformation and thereby retrieve another data set for creating,combining, transforming, etc. according to embodiments described hereinto supply the missing information or an approximation of the missinginformation.

In an exemplary embodiment, the system may generate the missinginformation. The system may record historical information, and/or haveaccess to additional information.

For example, for a moving vehicle (such as a boat or airplane), thesystem may include a data source having historical information oflocation, speed, direction, time, or a combination thereof. The systemmay also include a data source having information about the origin,location, destination, departure time, expected arrival time, orcombinations thereof. The system may use any combination of the dataavailable to project or estimate data that may be missing. In the eventthat system received present location data at a delay from real time,the system may be configured to estimate the present location of thevehicle based on any combination of the data it has available. Thesystem may use the last known location, the destination location, thetime at the last known location, the expected time to arrival, and thecurrent time to estimate a present location at the current time betweenthe last known location and the destination location. The system may usea plurality of last known locations along with their associated timestamps to determine a travel speed, direction, and an elapse time fromthe last known location to estimate a present location.

As another example, for a sensor feed, the received data set may onlyinclude the sensor data or may also include a time stamp. The locationinformation may be missing. The system may be configured to detect orreceive the location information for placement of the data sourceinformation within the virtual environment.

As another example, for the parcel location information, such as houselocations and parcel boundaries, the data source may include latitudeand longitude coordinates, but may be missing the elevation information.The system may therefore use, for example, the elevation mesh createdfrom the physical environment data source, such as described herein withrespect to step 308, to supply the missing elevation data for the givenlatitude/longitude coordinates provided within the given data set. Thismay occur, for example, when different data sets use information fromother data sets, such as illustrated by the dashed line between steps308 and 316 of FIG. 3 .

In an exemplary embodiment, if more than one data set is provided, thedata sets may be combined. The combination may include combining and/orfiltering the combined data set. For example, two data sets may overlayand produce redundant data information that may be filtered and/orremoved from the combined data set. Other data set combination andfiltering methods and techniques may also be used and are within thescope of the instant disclosure.

In an exemplary embodiment, the data set may be transformed to a usabledata set for generating virtual objects and/or for use in theaugmented/virtual reality rendering of the virtual objects. Thetransformation to a usable VR/AR data form may be combined or integratedinto one or more of the steps described herein, such as in creating auniform structure. The transformation may also be performed separately,such as before or after one or more other data transformation/analysissteps.

All modern 3-D graphics engines utilize triangles to render items.Vertices (X,Y,Z points) and triangles create the geometry that describesall required points of the digital surfaces. Specific “shader programs”then draw the digital surface and control its look, texture (imagesplaced upon it), how it interacts with lighting, etc. Device resourceoptimization avoids rendering obscured areas. Exemplary embodimentsdescribed herein may include transforming the data sets received about aphysical environment and tessellating the data to transform the data setinto a triangular data set for graphics rendering. For example,conventional physical environment data sets are generally taken in gridssuch as square or rectangular spacing. Each rectangular data set can bedivided into two triangles for triangular representation.

As illustrated in FIG. 3 at step 306, exemplary embodiments may includesimplifying the data set(s) by reducing the information within the dataset while maintaining an appropriate fidelity for a given application.

In an exemplary embodiment, Level of Detail (LoD) techniques may be usedto simplify a data set. LoD involves rendering the models at a fartherapparent distance from the “viewer” with less detail, saving systemresources while remaining unnoticeable to the user. This concept lendsitself to rendering of Digital Elevation Maps (DEMs). Typical DEMs(USGS's National Map Elevation data, NGA's worldwide DTED data, etc.)contain a series of regularly spaced points on a grid that contain allrequired elevation data. The dataset corresponding to locations closerto the point of interest, such as the user for rendering within theaugmented reality environment, include denser data sets or more points,such as all of the spaced points on a grid, while locations that arefurther away from the location of the point of interest may be lessdense data sets, such that the data corresponds to information atlocations having greater space between their points on the grid. In anexemplary embodiment, information very far away, such as out of sight ofa user at the point of interest, may not be retrieved or displayed atall. The point of interest may be provided to the system, such as by auser entry. The point of interest may be automatically achieved, such asby receiving location information from the system. The point of interestmay be achieved through communication through one or more other sensors,devices, sources, etc. For example, the information of the locationand/or orientation may be obtained at step 312 and provided at step 306to use to simplify the data set.

In an exemplary embodiment, Digital Elevation Map (DEM) simplificationhelps reduce system load. A “2-D Decimation” converts the DEM grid totriangles (2 triangles for 4 points), giving an equal density oftriangles across the surface. A “Simplified Triangle Mesh” reduces thenumber of triangles needed while retaining detail by combining trianglesacross all flat surfaces. For example, a nearest neighbor comparisonbetween adjacent and neighboring triangles can be determined. If thevariation between triangles is within a desired threshold, then theinformation within the triangle can be combined, thereby reducing thedata set.

In an exemplary embodiment, the transformed information may be taken atdifferent densities. Thereby, given data sets may be filtered to removeintervening values or averaging values over larger dimensional spaces.Such averaging or skipping data points may reduce fidelity. Therefore,effects and assessment on the resulting fidelity is preferred.

As illustrated in FIG. 3 at step 308, exemplary embodiments describedherein may create a geographical mesh representing a physicalenvironment that can be augmented by the system. The mesh may be avirtual mesh that may or may not be displayed to the user. The mesh maythereafter be used to transform and/or supply missing information withindata sets to be rendered in the virtual space and overlaid on a physicalenvironment.

FIGS. 4A-4D illustrate a mesh overlay in which a data set of elevationis rendered on a physical environment. The data set was received andrendered in a triangular format, such as seen in FIG. 4A. FIGS. 4B-4Cillustrate the same data set taken at different intervals and/oraverages such that the granularity and therefore fidelity of theinformation is reduced. FIG. 4A may illustrate the representation of adataset that has been transformed according to embodiments describedherein and retain a granularity and/or fidelity approximate to theoriginal data set. FIGS. 4B-4D may illustrate the representation of adataset that has been simplified according to embodiments describedherein and may include reduced granularity in the dataset whilemaintaining a fidelity of a desired level relative to the raw dataset.

FIG. 4C illustrates an exemplary mesh in which the data fidelity ismaintained by combining areas of little transition to create largergranularity (lower density mesh) and maintaining the data set for areasof greater transition to create finer granularity (higher density mesh).For example, the lake portion of the physical environment does notchange elevation for significant portions of the surface, therebypermitting triangular representations of the elevation to be combinedand a few data points to define a larger geographic area, while thehills having greater transition are represented with greater densitysuch that greater data points are used to define the geographic area.

Determining whether the fidelity can be lowered, such as for areas oflittle transition between adjacent data points, can be achieved indifferent ways. For example, the system may directly compare adjacentdata sets to determine if the deviation is within a threshold. If it is,the system may average the data points or otherwise determine anappropriate value for the aggregated data point. In an exemplaryembodiment, the aggregated data point may be a weighted average, wherethe weight is related to a deviation between the value to be weighted inthe average and a value in an adjacent data point that is not weightedin the average. In this way the value that has a near neighbor thatchanges more aggressively can influence the average of the combined setmore than the next point that then transitions less aggressively. Incombining multiple adjacent data points, the system may compare the datapoints across the data set and then combine appropriate points that arewithin a given threshold to create the aggregated data point andcorresponding value of the aggregated data point. In combining multipleadjacent data points, the system may compare individual data pointssequentially against a given threshold and determine interim values forthe aggregated data point. The interim aggregated data point may then beused in the comparison for the next sequential data point. In combiningmultiple adjacent data points, the system may compare adjacent datapoints sequentially against the threshold and determine the aggregateddata point after the multiple adjacent data points have been identified,such that an average is taken from all of the identified multipleadjacent data points to be aggregated.

As another example of how the system may determine whether the fidelitycan be lowered, knowledge of identified attributes may be used. Forexample, lakes do not change elevation because of the nature of restingwater. Therefore, if the system either determines the presence of alake, such as through image recognition, or the system receivesinformation about the presence of a lake, such as through a data sethaving the boarder of a lake or through user inputs of identification oflakes, the system may lower the fidelity of the data set within theidentified area. The identified may be used to increase or lower thefidelity of the data set within the identified area. For example, ifmountains, hills, or other structure known to change elevation mayrequire a higher fidelity data set, verses the areas identified aslakes. As another example, information about roads may be provided at agiven fidelity because it is known that roads have a maximum elevationtransition given the limitations of the vehicles that use the roads.Therefore, road data may be given in a desired fidelity to capture thedesire level of detail given the known qualities of the data set.

Once the data sets are combined, transformed, and simplified, a digitalmesh is defined to create a local reference frame for the augmentedreality system related to the real world environment within which theaugmented reality system resides and used for viewing and/or augmenting.

Referring to FIG. 3 at step 314, the system also receives another datasource for information for rendering the virtual objects. Theinformation may relate to textual information, and/or relate to virtualgraphical representations to be overlaid on the view of the physicalenvironment. Similar to the datasets of 302 n, the dataset received forthe virtual objects may include one or more datasets in one or more dataconfigurations. Therefore, similar to steps 304-308, the system at step316 may combine, analyze, transform, simplify, or otherwise manipulatethe dataset to generate a usable dataset for generating the desiredvirtual objects. Therefore, the information received about the virtualobjects from the other data set(s) may be transformed such that it canbe rendered within the graphical environment of the augmented realitysystem. Exemplary transformations may include using the mesh toincorporate elevation or other missing data information into the datasource for the virtual object.

As seen in FIG. 3 step 312, the system may also determine the locationand orientation of the headset itself, and/or other desired viewpointfor rendering the virtual objects relative to the physical environment.The determination may be used to properly render and position thevirtual objects within the physical environment. The location andorientation of the headset may also be used to simplify the data set,such as in determining a level of fidelity and/or granularity for any ofthe one or more data sets for the physical environment and/or for thevirtual object, at steps 304, 306, and/or 316.

In an exemplary embodiment, the system may be initially calibrated suchthat its position and/or orientation is entered or determined by thesystem. Once known, the system may track location by sensors within orexternal to the headset. Exemplary embodiments permit the headset todetermine its own location and/or orientation based on inputs identifiedin its environment and/or from one or more systems communicating with orpart of the system. In an exemplary embodiment, the system may becalibrated by entering a GNSS location of the headset and directionalorientation. The system may include one or more sensors for determininglocation, and/or direction. The system may include a global navigationsatellite system (GNSS), local navigation position system, globalposition system (GPS), magnetometer, compass, accelerometer, vibrationsensor, tilt sensor, and combinations thereof to determine a locationand orientation of the system and/or track or determine a continuedlocation and/or orientation from an original location and/or position.The system may include external location tracking, such as, for example,ultrasound, acoustic, light, infrared, ultraviolet, radio frequency, orother signal detection for determining location and/or orientationrelative to one or more other known sources.

At step 310 of FIG. 3 , the system may generate and/or display a virtualobject for display in an augmented reality environment. The system mayuse any combination of data sets described herein for rendering anddisplaying the virtual objects.

In an exemplary use case, an exemplary embodiment of geopositioningsoftware ingests multiple types and classes of data sources. Theexemplary test case may use USGS 1/9 arc second (˜3 m nominal) postspacing, sub-centimeter resolution elevation data and OpenStreetMapvector data covering the same physical area. The information may becombined to create a comprehensive, reduced, fidelity mesh forrepresenting a reference frame of the physical environment. An exemplarymesh may define the topography of the physical environment. The vectordata may include additional information such as roads, power poles,lines, and several points of interest, but is configured only aslatitude and longitude information, without elevation data. Virtualobjects related to this information, such as in the informationassociated with the vector data (i.e. road names) or virtual objectsrepresenting the vector data (i.e. icons for lines, posts, etc.) may berendered or overlaid once the data is transformed and supplemented withthe mesh to provide the missing information from the data set. Thishighlights a typical technical challenge when merging multiple datasets. Exemplary embodiments may therefore manage all required dataconversions and renders a merged data set in an exemplary virtualreality or augmented reality display.

Exemplary embodiments may be used to create real-world positionalaccuracies comparable to those delivered by commercial internet mappingproviders.

Exemplary embodiments may provide novel and meaningful ways for userviewing and interaction within an augmented or virtual reality system. Areal-time view of georegistered information can be a powerful tool for auser, commercial, enterprise, and military. Although discussed herein interms of augmented reality systems, exemplary embodiments are equallyapplicable to virtual reality systems. Therefore, whenever augmentedreality or AR is referenced herein, virtual reality is intended to beincluded as well. Exemplary embodiments also do not require either an ARor VR environment, but may also be applicable to any display,manipulation, and/or ingestion of georegistered information through anydisplay system, such as on a conventional computer display or mobiledevice screen.

Exemplary embodiments described herein may include software configuredto run on a computer and/or VR/AR system to georegister datasetscorresponding to the VR/AR system and/or geographical image rendered byan electronic device. Exemplary embodiments may be used to support usersthat desire hands-free, eyes-out solutions.

FIG. 5 illustrates an exemplary data processing representation for theback end and real time processing according to embodiments describedherein. In an exemplary embodiment, the system includes a back end forreceiving, generating, storing, combining, augmenting, transforming,simplifying, or otherwise processing data according to embodimentsdescribed herein. Datasets in a form usable by the AR system maythereafter be cached on the AR headset and/or local system for use inrendering a real-time virtual object for an augmented reality view. Theheadset may use its dynamic positional and directional data to renderthe virtual object in real time and corresponding to the physicalenvironment that is being augmented.

Exemplary embodiments of the back end may generate any variety ofinformation based on any available or created data set(s). For example,the back end may include the raw data sets and/or the processing andmemory capabilities for combining, augmenting, transforming, simplifyingdata sets according to embodiments described herein, such as withrespect to the physical environment and/or the virtual object. The backend may also include the augmented reality usable data sets, such asthose having been processed, combined, filtered, and/or transformed. Asillustrated, exemplary data sets may include light detection and rangingdata sets for providing an elevation data set of a physical area. Otherdata sets may include digital elevation tiles, building footprints, roadand place data, or any other type of geospatial data.

This data may be made available to the AR headset via a networkinterface. The AR headset knows its location or the location of thedesired data, and may pull down the data that is relevant. The systemmay therefore be configured to read data, analyze or identify desireddata, and download and/or retrieve data relevant to the AR headset, suchas based on data related to location(s) proximate to the headset orwithin a predefine and/or user defined proximity distance. The data maythen be cached on the device, removing the need for constant networkaccess. If the data is in a format not suitable to the current displaymethod (for example, a highly detailed elevation map when only anoverview is current being viewed), the data may be converted on thedevice to the necessary display formats, with the high-resolution dataretained for future use in geolocation or other display scenarios.Alternatively, the system may be configured to realize the currentviewing setting and pull down a dataset and/or create a datasetcorresponding to the current display method. As an alternative, avariety of different expected formats can also be generated on theback-end, allowing for faster data transfer and less calculation, asneeded.

In an exemplary embodiment, data can be loaded from other AR headsets onthe same network, allowing for faster data transfer and less calculationwithout the need for a backend. In an exemplary embodiment, multipleheadsets may be used in conjunction for distributive storage,processing, etc.

FIG. 6 illustrates exemplary system configuration according toembodiments described herein. Exemplary embodiments of the systemdescribed herein may include an augmented reality system 1006. Theaugmented reality system may include a processor, memory, and/orcommunication interface to communicating with remote systems, such asserver 1003 and/or memory comprises databases 1005. Exemplaryembodiments may also include a computer, computers, an electronicdevice, or electronic devices. As used herein, the term computer(s)and/or electronic device(s) are intended to be broadly interpreted toinclude a variety of systems and devices including personal computers1002, laptop computers 1002, mainframe computers, servers 1003, mobilephone 1004, tablet, smart watch, smart displays, televisions, augmentedreality systems, virtual reality systems, and the like. A computer caninclude, for example, processors, memory components for storing data(e.g., read only memory (ROM) and/or random access memory (RAM), otherstorage devices, various input/output communication devices and/ormodules for network interface capabilities, etc. For example, the systemmay include a processing unit including a memory, a processor, aplurality of software routines that may be stored as non-transitory,machine readable instruction on the memory and executed by the processorto perform the processes described herein. Additionally, the processingunit may be coupled to one or more input/output (I/O) devices thatenable a user to interface to the system. By way of example only, theprocessing unit may receive user inputs via a keyboard, touchscreen,mouse, scanner, button, camera, hand gestures, voice command interfacesor any other data input device and may provide graphical displays to theuser via a display unit, which may be, for example, a conventional videomonitor, an augmented reality display system, or a virtual realitydisplay system. The system may also include one or more wide areanetworks, and/or local networks for communicating data from one or moredifferent components of the system. The system may receive and/ordisplay the information after communication to or from a remote server1003 or database 1005.

In an exemplary embodiment, the system may be configured to storesoftware and data sets directly on one or more display devices, such asthe AR headset. The system may download the requisite information forrendering virtual objects at the AR headset and/or system componentsproximate the AR headset. Therefore, the AR headset may be remote orseparated from the network such that local data stored or generateddirectly at or on components in local communication with the AR headsmay be used for rendering the virtual reality objects. The headsetsystem may therefore be used when access to remote servers or thenetwork are not available. The system may be configured to update datasources, data, or locally stored information when a connection isavailable. The updates may be two way, such that information received,generated, or used at the headset system may be communicated to a remotelocation for processing, storage, backup, viewing, etc., as well asupdates from remote locations used to update the information at thelocal headset system. The updates may also be configured to be runmanually and/or automatically. In an exemplary embodiment, the augmentedreality headset system may detect connection to a network. Exemplaryembodiments may therefore be used without interruption even whenconnects are sparse and have interruptions, while still providing themost real time information available to local AR headset.

Exemplary embodiments described herein may be used in any augmentedreality system to define a software library to enable a standardaugmented reality or virtual reality system to become a fully“geo-capable” device. By converting the user's local “flat spaceenvironment” into a world-wide (latitude/longitude/altitude or L/L/A)reference frame, exemplary embodiments of a geo-capable augmentedreality system including software visualizes where the user is lookingrelative to the real world. With exemplary embodiments of thegeo-positional software, an augmented reality device can displayreal-time georeferenced data sets to the user without having its own GPSsystem. Exemplary embodiments may also be used with systems having aGNSS, Inertial Navigation System (INS), or other positioning system. Inexemplary embodiments, the system may be configured to provide higherfidelity geolocation predictions.

Exemplary embodiments include a software library for storage andexecution at the display device for rendering virtual objects in realtime based on geo-positional data sets. Exemplary embodiments include astorage of one or more geo-positional data sets at a remote serverand/or database that may be downloaded and cached by an augmentedreality system. In an exemplary embodiment, the system is configured todownload only a partial data set of the geo-positional data setcorresponding to the data corresponding to geo-positions within a givenproximity to the display device and/or user and/or specified locationand/or region.

Exemplary embodiments may be configured to render virtual objects withrespect to information within a data set and/or use the data set toretrieve information from other data sources to provide additionalinformation or virtual objects for overlay within the augmentedenvironment. For example, an incoming data set may include aircraftinformation such as tail number and location. The system may beconfigured to retrieve the tail number and look up other informationabout the aircraft, such as its type, an image of the aircraft, itsdestination, its passenger list, etc. Exemplary embodiments may then usethis information to generate virtual objects for rendering in theaugmented environment.

Exemplary embodiments may also include rendering virtual objects and/orproviding user displays in different user display configurations. Thesystem may therefore include user input and/or interface devices toreceiving input from a user and/or for changing the user view of therendered virtual objects. For example, a user may select a view in whichvirtual objects are rendered in a 1:1 scale to their physicalenvironment. The user may also select to view in which a user mayessentially zoom in or zoom out to get an overview perspective ofadditional information. For example, a user may transition a selectedview to reduce visual clusters in a 1:1 scale by using a smallerproximity distance verses the overview or larger scale view with alarger proximity distance.

Exemplary embodiments may also include on the fly rendering or userinputs of virtual objects to be rendered and/or created as a dataset orincluded in an existing data set. For example, as a user is within thesystem and perceives the real time rendered virtual objects in theiraugmented reality environment, the user may also provide inputs to thesystem to generate additional virtual reality objects and/or to modify,alter, delete, or add to the virtual reality objects already rendered bythe system.

Exemplary embodiments may also permit the export of information from thesystem. For example, a user may capture a video and/or still imageduring use of the system that may include an overlay of the virtualobjects rendered during use. Such exported video, still frames, and/orsequences of images may be used for remote monitoring, collaboration,quality assurance, record retention, and/or other purpose. Therefore,exemplary embodiments may include a camera or other image capturingsystem, or other method of retrieving an image, and a system foroverlaying or altering the image to include a rendered virtual object.Exemplary embodiments may permit the one way and/or two way sharing,manipulation, real time display, etc. between the augmented realitysystem and another display, whether another augmented reality systemand/or other display system.

FIGS. 7A-7C illustrate an exemplary embodiment of a user display of anediting suite that enables integration of precision geo-registered 3Dinformation into video for a Utility Inspection application. FIG. 7Aillustrates an exemplary raw fee of an aerial video and/or still, andFIG. 7B illustrates an exemplary augmented display with geo-referencedKeyhold Markup Language (KML) load and adjusted metadata. Export ofstill frames and/or image sequences may be extracted and shared withother users and/or stored. FIG. 7C illustrates an exemplary userinterface including one or more user interface input/output displays formanipulating the display of one or more virtual objects displayed on theimage and/or to a user.

Although exemplary embodiments described herein are for generating localreference frames for use with an augmented reality system, embodimentsand use of the system described herein are not so limited. Exemplaryembodiments may also include display of physical environments augmentedby virtual objects in other display systems, such as virtual reality orscreen display systems.

Referring to exemplary embodiments of FIGS. 7A-7C, a user may useembodiments of the system described herein to view a physical locationand/or virtual objects augmenting the physical location as describedherein. The image of FIGS. 7A may be a view of a user from a helicopteror other elevated location to get a perspective view of an area. Thesystem may include a camera for capturing an image or a series ofimages, such as in a video, of the physical environment. The camera mayinclude location and orientation information similar to the augmentedheadset embodiments described herein. The system may use thatinformation for locating, orienting, positioning, directing, orotherwise rendering virtual objects for presentation thereof. FIG. 7Btherefore may represent a view of a user in real time of the virtualobjects augmenting the physical view of the user, such as through anaugmented reality system. The FIG. 7B view may also represent thedisplay of the physical environment with augmentation by virtual objectsthrough other user interface displays. The system may be configured towork in either configuration or simultaneously such that a user may viewthe physical environment augmented by virtual objects in real time ordelayed through a virtual reality system, augmented reality system,and/or screen display system.

In an exemplary embodiment, the location and/or orientation of thephysical environment such as viewed through the augmented reality systemmay be offset from the environment captured through the camera or otherrecording method. For example, the user may have an augmented realityheadset according to embodiments described herein in which the user iswithin a cabin of a helicopter, while the camera for capturing imagesand/or video is positioned on an exterior surface of the helicopterbody. The replay between the captured images and the rendered virtualobjects may therefore be inaccurate during reproduction for presentationon a display other than the augmented reality headset. The system maytherefore include a video/image editing suite to calibrate the capturedimages/video with the rendered virtual objects.

Exemplary embodiments may include a specialized video editing suite thatadds an AR type overlay to airborne video or images. For example,airborne video gathered during electrical infrastructure inspectionevents may be used as a baseline environment for which virtual objectsare rendered thereover. The baseline video may be associated with asynchronous metadata file that enables the precision georegistration ofexternal digital information sources. In addition to allowing theinsertion of graphical information via a KML/KMZ file (or other datasource as described herein), exemplary embodiments allow the precisealignment of the digital information by providing adjustment relative tothe underlying coordinate system. This capability enables the user toeliminate or reduce the earth/imagery alignment errors typically seen ingeo-registered airborne video footage. By integrating three sources ofinformation (for example, but not limited to, clean video, sensormetadata and a KML/KMZ file), exemplary embodiments described herein maybe used to create usable still or video imagery featuring a synthetic,georeferenced overlay (the KML/KMZ information). KMZ file is a zipped orcondensed KML file.

Exemplary embodiments include a simple user interface that may allow theoperator to precisely position georeferenced overlays onto underlyingimagery. Exemplary embodiments include a tool kit that can permitrequired adjustments to create accurate alignment of a virtual overlayto an imagery file. Exemplary embodiments may include compensationfunctions for different alignment errors seen in georeferenced imagery,such as, without limitation: simple commands to load the media file,metadata set and/or KML information; traditional play/pause/scrubbuttons to control the video; easy-to-use sliders to control the virtualobject's Altitude, Heading, Pitch, and Roll relative to the underlyingimagery; sensor field of view slider to control the relative size of theoverlay, such as relative zooming; slider interface that allows precisealtitude corrections relative to the base Digital Elevation Map; frameoffset command allows user adjustment of the metadata file timing ifnecessary; and any combination thereof. Exemplary embodiments may befully customizable and can be tuned to meet specific application needs.

FIG. 8 illustrates exemplary embodiments used to show future buildinglocations and project phases.

Referring back to FIG. 2 as an exemplary specific use case, exemplaryembodiments may, be used to visualize, in real-time, local aircraftbased on a geo-position-based display of Automatic DependentSurveillance-Broadcast (ADS-B) information. An air traffic displaysoftware (such as, for example, Augmntr's HoloAirTraffic) displays eachADS-B enabled aircraft in the vicinity (set as a 50 km radius around theuser) as a digital icon, and shows the target's tail number, altitude,and range (relative to the user). Exemplary embodiments permit a user toconfigure a proximity radius, distance, or location of interest anddisplay virtual objects in relation to select physical objects in theconfigured proximity. In an exemplary embodiment, the proximity maychange based on a selected view to reduce visual clusters in a 1:1 scaleby using a smaller proximity distance verses the overview or largerscale view with a larger proximity distance. The specific aircraft canbe ‘selected’ to access additional levels of information, including aphotograph of the actual airplane (if available) and the ownershipinformation. Exemplary embodiments use the aircraft's tail number (fromthe ADS-B data packet) to query open-source web-based data servers foradditional information on the aircraft in question. Embodiments of thistechnique can be applied to any type of vehicle, vessel or otherGNSS-tracked asset assuming the data set is available to draw from.

Exemplary embodiments used ADS-B for aircrafts, as this is a low-latencysystem that is freely accessible. This same technique can be applied toviewing ships, or other vessel, with their AIS transponder system.Likewise, this could be adapted for example for law enforcement, fireservices, Customs & Border Patrol, airports/airfields, large utilityinstallations, shipping ports, or similar entities with fleets ofvehicles. As long as the individual vehicle is reporting its locationvia some method, exemplary embodiments described herein may be used todisplay this information in an augmented reality fashion, both as anoverview view and in a 1:1 implementation. Exemplary embodiments,therefore, are configured to access a suitably-defined data stream whereeither the vehicle (or other object) self-reports it's ID number,location and time stamp, or this same data is provided by an externalmonitoring method.

Exemplary applications that may benefit from exemplary embodimentsdescribed herein, include, without limitation: an AR device-basedsolution that renders real-time geo-registered information; AR and/or VRcontrol of external sensor nodes via voice interface or other commandmethods such as hand gestures, etc.; rendering large, complex data setsinto both a georegistered, 1:1 scale and overview-scale displays toprovide novel imaging solutions; ability to switch from overview-scaledisplay to a specific location in the 1:1 scale mode by looking at thedesired point of placement and issuing the command to switch view modes;ability for the 1:1 viewing mode to simulate the presence of a physicallocation by displaying appropriate digital information representingitems that are present at that actual location (icons representingroads, buildings, landmarks, etc.).

Exemplary embodiments of the simulated 1:1 mode can be utilized at anylocation to allow the user to get the impression of what would be seenat the actual corresponding real-world location. Exemplary embodimentsmay also permit users to create incorporate text comments, text data,planned path routings, insert shape files, or other intended referenceitems in the overview view mode that are then displayed at the selectedlocation to the user when in 1:1 view mode, either in the simulatedposition or at the physical location on the earth.

Exemplary embodiments are designed to be hardware-agnostic and are nottied to a specific AR or VR product, brand or service, thereby allowingthe customer to use appropriate AR/VR technology evolutions as theymaterialize.

Exemplary embodiments of the system described herein can be based insoftware and/or hardware. While some specific embodiments of theinvention have been shown the invention is not to be limited to theseembodiments. For example, most functions performed by electronichardware components may be duplicated by software emulation. Thus, asoftware program written to accomplish those same functions may emulatethe functionality of the hardware components in input-output circuitry.The invention is to be understood as not limited by the specificembodiments described herein, but only by scope of the appended claims.

Although embodiments of this invention have been fully described withreference to the accompanying drawings, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of embodiments of this invention as defined bythe appended claims. Specifically, exemplary components are describedherein. Any combination of these components may be used in anycombination. For example, any component, feature, step or part may beintegrated, separated, sub-divided, removed, duplicated, added, or usedin any combination and remain within the scope of the presentdisclosure. Embodiments are exemplary only, and provide an illustrativecombination of features, but are not limited thereto.

Exemplary embodiments of the systems described herein may be used invarious application such as, without limitation a 3D Enriched UrbanTerrain that may develop and deploy a suite of tools and workflows torapidly and cost-effectively collect and process accurate, measurable,updatable interactive displays of maps/models of geometry, materials andfunctions of vertical and horizontal infrastructure.

Exemplary embodiments of the systems described herein may be used invarious application such as, without limitation an Integrated VisualEnsemble that may include a Head Mounted Display System that providesenhanced Situational Understanding capability to the Warfighter byintegrating targeting imagery, geolocation information, situationalawareness and mission command information without increasing Soldiercognitive burden.

Exemplary embodiments of the systems described herein may be used invarious application such as, without limitation a Mission PlanningTechnologies for Small Units that may include tools that enablesynergistic and optimized mission planning at the company and belowlevel during one or more phases of the operation.

Provided herein is a simple case study to examine the challenges ofcurrent augmented/virtual reality systems and overlaying datasets insuch environments. Difficulty in overlaying datasets in an augmentedreality system may be due to the terrain and chain in elevation over anarea. The exemplary use case includes an available high-resolutionraster elevation data (derived from an also-available LIDAR pointcloud), and interesting vector data. The goal of this example case studyis to incorporate both the raster elevation data and vector data into aMicrosoft HoloLens as a representative mobile augmented reality headsetdevice. This initial case study example does not incorporate the rawLIDAR point cloud into the HoloLens, although this could be accomplishedas an alternative embodiment.

All data used for this case study came from open sources. The rasterelevation data's source is the United States Geological Survey (USGS)and features 1/9 arc second (˜3 m nominal post spacing, sub-cm elevationresolution) data for an area 683 m×875 m (0.42 mi×0.54 mi), totaling˜255 KB of raw elevation data. The vector data is from OpenStreetMap andcovers the same physical area. This vector data contains roads, powerpoles and lines, and several other points of interest. All items fromOpenStreetMap contain only latitude and longitude information, with noaltitude data available. This presents a representative example of atechnical challenge described herein in which data sets are incompleteto fully render an overlay in a three dimensional environment.

Exemplary embodiments include software configured to ingests severaldata formats. Utilizing this software asset, exemplary embodiments arealso configured to convert the source raster elevation data from IMGformat to ESRI Gridfloat, and the OpenStreetMap vector data to KML.WGS84/EGM96 was selected as the common datum and converted all data setsas necessary, although other common system may be used. This conversionpermits uniformity and coordinate transferability across the respectivedata sets. All individual coordinates remained unaltered with no manualposition adjustments applied. This process maintained the relativepositioning of all points within the data sets. The only adjustmentsapplied to the vector data was manual styling of relevant icons, thechoice of vibrant colors, and selection of reasonable line widths forobjects, such as, for example, streets and highways. Exemplaryembodiments of the software renders these data sets as a common 3-Drepresentation containing both raster elevation data and the vector dataset. The software features display options allowing the data view aseither wireframe format, or with a texture applied.

Given the real-world scale of these types of data sets, exemplaryembodiments may include having a “overview mode”. Exemplary embodimentsof the software may provide a configurable viewing scale that includesboth a 1:1 display and an overview mode that permits an easy overview oflarge areas. Exemplary embodiments of the overview display mode may beuseful and beneficial for tasks such as mission planning, areaoverwatch, asset placement monitoring, etc.

Processor demand and complex graphics display activities consume bothbattery power and can degrade the video and processor bandwidthnecessary for 60 fps viewing. Therefore, advanced techniques to reducerendering complexity as described herein, while retaining data setintegrity and detail, may be incorporated to deploy rich augmentedreality 3-D experiences to wearable AR devices. For example, Level ofDetail (Lod) techniques, DEM simplification, and/or other methods may beused to provide a desired frames per second viewing while compensatingfor battery power and bandwidth.

Exemplary embodiments may include putting together all the capabilitiesdescribed herein or any combination thereof. This example combines bothstatic data sets (a DEM and a vector data set) along with real-timemovers reporting positions via transponder feeds. This demonstrationintegrates these real-time moving geo-registered objects (ADS-B enabledaircraft) into the baseline DEM (a 2-D decimated raster elevation dataset as described herein, such as represented in FIG. 2 ) and renders anAR display on the HoloLens. All aircraft positions are appropriatelyregistered relative to the underlying elevation data set and to eachother. FIG. 9 illustrates a 50 km area around Las Vegas, NV due to theinteresting map elevation features and a high density of variousaircraft types at various altitudes. FIG. 9 illustrates an exemplaryoverview display configuration according to embodiments describedherein. FIG. 9 illustrates an exemplary HoloLens showing the “AR BlueForce Tracker” demonstration. This shows real-time moving aircraftgeo-registered to underlying DEM and vector data display around the LasVegas area.

As an example of applications of embodiments described herein, a missioncommander may use exemplary systems and methods described herein formonitoring troop movements and asset positions, in a real-time 3-D ARdisplay. The underlying map or battlefield surface could featurenumerous disparate data sources, all converted, aligned and displayed ina method according to embodiments described herein. Utilizing exemplarywork flows described herein enable the creation of such an augmentedreality display.

Referring back to the “overview” viewing mode, exemplary embodiments maybe of significant benefit for tasks such as asset placement, areaoverwatch, and infrastructure identification tasks, etc. Exemplaryembodiments may be used for infrastructure identification and may rendergeoregistered AR display technologies. Exemplary embodiments may displayvirtual objects that may be obscured by physical features, such as thelandscape, buildings, or other object. For example, a system may displaya line of utility poles in which the augmented reality display stillshows the poles, their identity and location, even though physical polesmay be obstructed by hills or other features.

Imagine an addition to this scenario where a power service crew vehicle,equipped with a location transponder, was in the scene. In thissituation, the AR display would show the work vehicle position inaddition to the pole locations in both the 1:1 and overview viewingmodes. This gives a new method of managing the infrastructure and themaintenance assets. For example, referring to FIGS. 4A-4B, if a workvehicle were behind a hill and not physically visible to a user of theaugmented reality system, the vehicle may be identified and displayed asa virtual object, such as in text, symbol, or other representation, tothe user at their location to indicate its presence and location.Exemplary embodiments of software described herein can show any sort ofinfrastructure (communication lines, plumbing, power lines, airportrunway lights, etc.) if there is geo-registered data connecting theasset's physical location to geographic coordinates.

Various industrial, utility installation, law enforcement and securityservice applications may incorporate exemplary embodiments herein wherelarge areas with distributed assets are actively monitored and/ormanaged. The underlying map/display surface could feature many disparatedata sources, all converted, aligned and displayed much like thisdemonstration. The AR headset delivers real-time unit status andlocation information via the holographic display. With both 1:1 viewingand the overview mode, the user can control the desired viewing mode togain better overall Situational Understanding of all assets andpersonnel.

As another use-case example, exemplary embodiments may be used for othersituational understanding activities at a neighborhood destroyed by theTubbs Fire, which started on Oct. 9, 2017 in Sonoma County, Calif. Thisfire destroyed a reported 5,643 structures and burned over 36,000 acres.In this Real-World test, the augmented reality system displayedgeo-registered property parcel data and revealed the location, addressand property parcel information of homes destroyed by the fire in CoffeyPark. By viewing holographic icons representing the former structures,the user is able to establish personal orientation relative to buildingsand landmarks no longer present. One can conceive that a geo-capableaugmented reality system could be of substantial benefit during rescueand recovery events following large-scale disasters, or in othersituations where traditional reference landmarks are unavailable. ARrepresentation of landmarks no longer in existence delivers SituationalUnderstanding. FIGS. 1A-1C illustrate this exemplary use case in whichvirtual objects in the form of cubes denote former home locations.

When used in this specification and claims, the terms “comprises” and“comprising” and variations thereof mean that the specified features,steps or integers are included. The terms are not to be interpreted toexclude the presence of other features, steps or components.

The features disclosed in the foregoing description, or the followingclaims, or the accompanying drawings, expressed in their specific formsor in terms of a means for performing the disclosed function, or amethod or process for attaining the disclosed result, as appropriate,may, separately, or in any combination of such features, be utilized forrealizing the invention in diverse forms thereof.

1. A method, comprising: receiving a data set; converting the data setto a common reference frame; and generating a virtual object based onthe data set in the common reference frame.
 2. The method of claim 1,wherein the received data set is missing information in order to renderthe virtual object in three dimensions.
 3. The method of claim 2,further comprising supplying the missing information and rendering thevirtual object in three dimensional space through an augmented realitydisplay system.
 4. The method of claim 1, further comprising receiving asecond data set relating to a physical environment, wherein the firstdata set comprises missing information and the second data set comprisesinformation related to the missing information of the first data set. 5.The method of claim 4, wherein the second data set comprises elevationof a ground level of the physical environment.
 6. The method of claim 5,further comprising converting the second data set into the commonreference frame.
 7. The method of claim 6, further comprisingsimplifying the second data set by reducing a resolution of the dataset.
 8. The method of claim 7, further comprising generating a meshmapping of ground level elevation based on the second data set.
 9. Themethod of claim 8, further comprising using the mesh mapping to supplythe missing information.
 10. The method of claim 1, wherein thereceiving the first data set comprises having a first data set that isgeo-registered.
 11. The method of claim 1, further comprising receivinga location and orientation of an augmented reality system for displayingthe virtual object, and displaying the virtual object based on thegeo-registered data set.
 12. The method of claim 11, further comprisingreceiving the data set in real time for real time generation of thevirtual object.
 13. A system for displaying real-time, geo-registereddata in an augmented reality environment, comprising an augmentedreality display system, one or more processors, memory, andnon-transitory machine readable instructions that when executed by theone or more processors are configured to: receive in real timegeo-registered information; transform the geo-registered informationinto a data set to be displayed by the augmented reality system; anddisplay a virtual object on the augmented reality display system basedon the real time geo-registered information.
 14. The system of claim 13,wherein real time geo-registered information is missing information forthe real time display of the virtual object in three dimensions and thenon-transitory machine readable instructions that when executed by theone or more processors are further configured to generate the missinginformation and display a real time display of the virtual object inthree dimensions on the augmented reality display system.
 15. The systemof claim 14, wherein the non-transitory machine readable instructionsthat when executed by the one or more processors are further configuredto simply the real time geo-registered information.